Variable-rate source coding theorems for stationary nonergodic sources

~ The source coding theorem and its converse imply that the optimal performance theoretically achievable by a Axedor variable-rate block quantizer on a stationary ergodic source equals the distortion-rate function. While a Axed-rate block code cannot achieve arbitrarily closely the distortionrate function on an arbitrary stationary nonergodic source, we show for the case of Polish alphabets that a variable-rate block code can. We also show that the distortion-rate function of a stationary nonergodic source has a decomposition as the average over points of equal slope on the distortion-rate functions of the source's stationary ergodic components. These results extend earlier Anite alphabet results. I . INTRODUCTION In [l], Shields et al. show that for any stationary nonergodic finite alphabet source, the distortion-rate function D(R) equals the infimum of the average of the distortion-rate functions of the source's stationary ergodic components, where the average is taken over points on the component distortion-rate functions whose rates average to at most R. The achievability of this bound by variable-rate block codes is shown in [Z]. We extend these variable rate quantization results from finite alphabets to complete separable metric spaces, or Polish alphabets. We employ a simplified variable-rate and variabledistortion using a Lagrangian formulation. 11. RESULTS Let ( A m , B m , p , T ) be a stationary dynamical system with Polish alphabet A. That is, let A be a complete separable metric space, let B be the Bore1 o-algebra generated by the open sets of A, let A" be the set of one-sided sequences z = (21, zz, . . .) from A, let B" be the o-algebra of subsets of A" generated by finite-dimensional rectangles with components in 8, let T be the left shift operator on A", and let p be a measure on the measurable space (Am,Bm), stationary with respect to T. Now let p ( z l , yl) < 00 be a fed-vdu+ nonnegative distortion measure for 21 E A, y~ E A, where A is an abstract reproduction alphabet. Assume that p ( z ~ , y ~ ) is continuous in z1 for each y1 E A and that there exists a reference letter y; such that E,p(Xi, Y;) < 00. Define p ( z N , a r N ) = Cf, p?,,y,). Finally let Q be a variable-rate block quantizer with blocklength N . That is, let Q be a map from A N onto some finite or countable set of codewords {yN} C AN composing a codebook C = {(yN,IyNI)) in which each codeword yN has an associated variable-length binary description, with length denoted l ~ ~ [ . The description lengths must satisfy the Kraft inequality xyNEC 2 4 ' 1 I 1. 'This material is based upon work partially supported by an AT&T; Ph.D. Scholarship, by a grant from the Center for Telecommunications at Stanford, and by an NSF Graduate Fellowship. The optimal performance theoretically achievable by any variable-rate block quantizer is the operational distortion-rate function 6"(R,p) = infN6Lr(R,p), where 6gr(R,p) is the N t h order operational distortion-rate function 6Lr(R,p) = i;f { ; E " p ( X N , Q ( X N ) ) ; iEpIQ(xN)I IE ] . Here, the infimum IS taken over all variable-rate block quantizers Q with blocklength N . We contrast this with the optimal performance theoretically achievable by fixed-rate block quantizers, 6fr(R, P ) = infN &(R, p ) , in which &(R, p ) is defined as 6Lr(R,p) but with the infimum taken over all fixed-rate block quantizers with blocklength N . The Shannon distortion-rate function is defined similarly, as D(R,p) = infNDN(R, p ) , where D N ( R , ~ ) is the Nth order distortion-rate function DN(R, p ) = i;f { ?E,,p(XN, 1 Y N ) : k I p U ( X N ; Y N ) I R } . Here, U is a conditional probability or test channel from A N to AN defining, with p , a joint probability or hookup pu on X N and Y N , and I is the mutual information. I t is well-known that both D(R, p ) and 6fr(R, f i ) are convex in R [3]; SVr(R,p) is likewise convex in R, by a timesharing argument. Hence 6"(R,p) and D ( R , p ) can be characterized by their support function& [4, p. 1351 the weighted operational distortion-rate function !(A, p ) = infR [6"(R, p ) + XR] and the weighted Shannon distortion-rate function L(X, p ) = infR[D(R, P ) + XR] . The source coding theorem and its converse imply that when p is ergodic, 6"'(R,p) = &(R,p) = D ( R , p ) for all R 2 0 (and hence P ( X , p ) = L ( X , p ) for all X 2 0). When p is nonergodic, let { p = : z E A") denote the ergodic decomposition of p. The ergodic decomposition exists since A Polish implies ( A , B ) standard [5, Theorem 3.3.11, and hence (Am, 8") is standard [5, Lemma 2.4.11 which gives the desired property by [5, Theorem 7.4.11. The main results of this paper are that under the conditions given above Theorem 1 ((A, p ) = Theorem 2 L ( X , p ) = s L ( X , p = ) d p ( z ) VX > 0, Theorem 3 C ( X , p ) = L(X,p ) VX 2 0, and hence hVr(R,p) =D(R, p ) VR 2 0.!(A, p.)dp(z) VX 2 0, REFERENCES[l] P. C. Shields, D. L. Neuhoff, L. D. Davisson, and F. Ledrappier, "The distortion-rate function for nonergodic souTces,I1 TheAnnals of Probability, 6(1):138-143, 1978.(21 A. Leon-Garcia, L. D. Davisson, and D. L. Neuhoff, "New re-sults on coding of stationary nonergodic sources," IEEE Trans-actions on Informafion Theory, 25(2):137-144, March 1979. [3]R. M. Gray, Entropy and Information Theory, Springer-Verlag,New York, 1990. [4] D. G. Luenberger, Optimization by Vector Space Methods, JohnWiley and Sons, New York, 1969. [5]R. M. Gray, Probability, Random Processes, and Ergodic Prop-erties,Springer-Verlag, New York, 1988. 0 7803-2015-8/94~S4.00(01994 IEEE266 -